branch-4.1: [fix](cast) Declare Variant casts to non-Variant targets nullable by eldenmoon · Pull Request #70363 · apache/doris

eldenmoon · 2026-11-07T04:50:03Z

What problem does this PR solve?

Cherry-pick of #70359 (adapted for branch-4.1) plus the BE scan fix proposed on master in #70365.

Problem Summary: backport the Variant-to-non-Variant nullable fix to branch-4.1 and retain the resulting null tests during scan normalization. A NOT NULL Variant column can contain Variant nulls or values that cannot be converted to a scalar. The existing BE CAST can return SQL NULL, but Cast.castNullable only accounted for Variant V2 and legacy casts to JSON.

This causes both wrong optimizer rewrites and execution contract failures: <=> NULL becomes FALSE, COALESCE/NVL lose their fallback, and direct CAST queries can fail with a declared non-nullable / actual nullable mismatch.

End-to-end validation exposed a second issue: after correcting FE metadata, scan normalization still replaced CAST(v AS INT) IS NULL with a null test on the source Variant column. This loses NULLs introduced by conversion; IS NOT NULL can retain the wrong rows.

Fix

Reuse the existing Cast.castNullable rule to declare Variant-to-non-Variant casts nullable for both representations. Identity casts retain input nullability; no optimizer-specific nullable rule is added.
Keep NULL tests over casts as expressions through the existing scan pushdown rejection path. The scan eligibility check only admits casts from Variant here, so ordinary slot NULL predicates keep their existing path.
Add focused FE optimizer coverage and legacy-Variant SQL regression for projections, filters, COALESCE/NVL, nullable inputs, incompatible values, identity casts, an aliased CTE, and all-Variant-null roots. The representation-specific SQL golden is explicitly skipped on V2; V2 nullable inference remains covered by CastTest.

Validation

Before the fix: FE tests reproduce the wrong nullable flag, FALSE folding of <=> NULL, and removal of conditional fallbacks. SQL fails with a nullable column mismatch. With the FE fix alone, scan filters still incorrectly return no rows; the captured projection results identify the rows that should survive.
After the fix: 59 FE unit tests passed (CastTest, CheckCastTest, NullSafeEqualToEqualTest, SimplifyConditionalFunctionTest, VariantCastNullableTest).
FE build with Checkstyle and BE ASAN build passed using build.sh.
Regression goldens were generated by the repository runner and inspected. Normal comparison: test_variant_cast_nullable and existing test_variant_cast passed (2 suites).
C++ formatting: repository wrapper with clang-format 16. Static analysis: attempted after the ASAN build; after correcting the toolchain resource directory, analysis is still blocked by the pre-existing unmatched NOLINTEND in be/src/core/types.h:577. Reported source warnings are outside the added lines; this is not a clean clang-tidy pass.

Compatibility and performance

BE CAST conversion semantics, storage formats, and Thrift definitions are unchanged. The fix corrects metadata, optimizer simplification, and filtering against the existing conversion result. Existing legacy Variant conversion quirks are preserved by the golden tests.

The FE change adds no per-row conversion work. Unsafe NULL-predicate pushdown over a Variant cast is replaced by expression evaluation, which may scan/evaluate more rows for those predicates; ordinary source-column NULL pushdown is unchanged. No performance benchmark is claimed. No new public API or dependency is added.

Release note

Fix incorrect NULL predicates, conditional-function simplification, and nullable result type mismatches when casting Variant values to non-Variant types with legacy Variant execution.

Check List (For Author)

Test: FE Unit Test: 59 passed; SQL regression: 2 suites passed; FE/BE ASAN builds passed
Behavior changed: Yes. Preserve NULL/fallback results and filter rows according to the CAST result; conversion semantics are unchanged.
Does this need documentation: No

Update: explicit Variant check + NOT NULL-column regression

Related BE fix on master: [fix](be) Keep null tests over Variant casts in the scan expression #70365 (this branch carries the same change, adapted).
The scan guard now checks the cast source type explicitly (Variant) instead of skipping every CAST_EXPR. Behaviour is unchanged for the reachable shape (only Variant casts pass is_valid_push_down_cast), but the condition is self-documenting and no longer relies on that implicit precondition.
Added variant_p0/test_variant_cast_isnull_pushdown: CAST(v AS INT) IS NULL, IS NOT NULL and <=> NULL on a VARIANT NOT NULL column. The existing is_null_filter only covered CAST(v AS STRING), where every row is non-null, so it could not catch a removed guard. The golden was generated by the regression runner. Without the guard the new test returns no rows for IS NULL / <=> NULL.

hello-stephen · 2026-11-07T04:50:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2026-11-07T05:09:57Z

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#70363
commit: 43ccf7d0a4e7df3e9dc8d566dfb209556decfedb
base: 3f4002e778bd4bdac135b349f349e5ed4d7a5093
reviewed_at: 2026-11-07T13:09+08:00
reviewer: eldenmoon
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

regression-test/suites/variant_p0/test_variant_cast_nullable.groovy:19 — This regression golden is intentionally limited to legacy Variant execution; V2 nullable inference remains covered by CastTest. The prior representation-mismatch finding is fixed at this head.
This was a static review. The author's final log records the two targeted SQL suites passing; reviewers did not execute builds or tests. Clang-tidy remains blocked by the pre-existing unmatched NOLINTEND in be/src/core/types.h:577 and is not a clean pass.

_{Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.}

eldenmoon · 2026-11-07T06:53:05Z

Update in this branch:

The scan guard now checks the cast source type explicitly (Variant) instead of skipping every CAST_EXPR; behaviour is unchanged for the reachable shape, but the condition no longer relies on the implicit precondition in is_valid_push_down_cast.
Added variant_p0/test_variant_cast_isnull_pushdown covering CAST(v AS INT) IS NULL, IS NOT NULL and <=> NULL over a VARIANT NOT NULL column (the existing is_null_filter used a STRING cast where every row is non-null, so it could not catch a removed guard).
Data note: the new case uses only '1', '"x"' and '[1,2]'. {"a":1} is avoided because 4.1 casts it to 0 while 4.2 casts it to NULL; string and array values are NULL on both branches, so the golden is branch-agnostic.
The same BE fix is proposed on master in [fix](be) Keep null tests over Variant casts in the scan expression #70365.

Local status: the new case and the existing test_variant_cast / test_variant_cast_strict_mode pass on a 4.2 build; the 4.1 golden of test_variant_cast_nullable is unchanged.

eldenmoon · 2026-11-07T07:01:02Z

run buildall

hello-stephen · 2026-11-07T08:59:51Z

BE UT Coverage Report

Increment line coverage 40.00% (2/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.86% (27332/44181)
Line Coverage	46.58% (285464/612871)
Region Coverage	42.16% (225786/535578)
Branch Coverage	43.76% (105281/240583)

hello-stephen · 2026-11-07T09:54:37Z

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

eldenmoon · 2026-11-07T10:33:49Z

run feut

hello-stephen · 2026-11-07T13:17:06Z

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

eldenmoon · 2026-11-07T19:35:37Z

run cloud_p0

eldenmoon · 2026-11-07T19:35:41Z

run external

eldenmoon · 2026-11-07T21:16:08Z

run cloud_p0

eldenmoon · 2026-11-07T21:16:10Z

run external

eldenmoon · 2026-11-08T02:16:16Z

run cloud_p0

eldenmoon · 2026-11-08T02:16:19Z

run external

eldenmoon · 2026-11-12T02:05:03Z

run cloud_p0

eldenmoon · 2026-11-12T02:05:08Z

run external

### What problem does this PR solve? Related PR: apache#70359 Problem Summary: Legacy Variant casts may return SQL NULL for NOT NULL inputs, but their non-nullable FE declaration allowed null-safe predicates to fold to FALSE and removed COALESCE/NVL fallbacks. Apply the shared nullable rule to both Variant execution representations, as in apache#70359. ### Release note Fix NULL predicate and conditional-function results, and nullable column type mismatches, for casts from NOT NULL Variant values. ### Check List (For Author) - Test: Unit Test - 59 focused FE unit tests passed; pre-fix CAST and optimizer tests failed as expected. SQL regression validation is recorded in the test commit and PR. - Behavior changed: Yes. Correct nullable declarations prevent invalid rewrites or result type mismatches; BE conversion semantics are unchanged. - Does this need documentation: No

…lization ### What problem does this PR solve? Related PR: apache#70359 Problem Summary: after fixing Variant CAST nullable metadata, a filter such as CAST(v AS INT) IS NULL can still lose rows. Scan normalization strips the Variant cast and substitutes a NullPredicate on the source column, although the cast can introduce SQL NULL for a non-NULL stored value. Keep null tests on casts as expressions using the existing pushdown rejection path. IS NOT NULL uses the same path. CAST conversion semantics and storage are unchanged. ### Release note Fix missing or extra rows from NULL filters over Variant casts. ### Check List (For Author) - Test: ASAN build.sh --be; focused SQL regression generation passed - Behavior changed: Yes. Evaluate the cast's NULL result before filtering. - Does this need documentation: No

…acks ### What problem does this PR solve? Related PR: apache#70359 Problem Summary: cover legacy Variant-to-non-Variant CAST nullable inference, optimizer preservation of IS NULL / <=> NULL and COALESCE / NVL, and scan filtering on CAST results. Include nullable inputs, incompatible values, identity casts, aliases, and all-Variant-null roots. Limit the representation- specific SQL golden to legacy Variant execution. The baseline FE tests fail on nullable inference and simplification; the baseline SQL fails with a nullable mismatch, and the FE-only fix still loses rows in scan filters. ### Release note None ### Check List (For Author) - Test: 59 FE tests passed; test_variant_cast_nullable and test_variant_cast passed with ASAN BE (2 suites); golden generated by run-regression-test.sh - Behavior changed: No - Does this need documentation: No

Make the scan normalization guard check the cast source type explicitly (Variant) instead of skipping every cast, and add the `variant_p0/test_variant_cast_isnull_pushdown` regression: `CAST(v AS INT) IS NULL`, `IS NOT NULL` and `<=> NULL` over a `VARIANT NOT NULL` column. Without the guard the `IS NULL` / `<=> NULL` filters return no rows, because the scan rewrites the null test on the cast into `v IS NULL` and drops the non-NULL rows whose cast result is NULL.

eldenmoon · 2026-11-12T02:11:46Z

run buildall

eldenmoon · 2026-11-12T14:45:53Z

run compile

eldenmoon · 2026-11-12T14:45:58Z

run beut

eldenmoon · 2026-11-12T21:26:29Z

run compile

eldenmoon · 2026-11-12T21:26:37Z

run external

eldenmoon · 2026-11-12T21:26:44Z

run cloud_p0

eldenmoon · 2026-11-13T02:20:56Z

run p0

eldenmoon · 2026-11-13T02:21:01Z

run external

eldenmoon · 2026-11-13T02:21:06Z

run beut

eldenmoon · 2026-11-13T02:21:11Z

run feut

eldenmoon · 2026-11-13T02:21:16Z

run external

eldenmoon · 2026-11-13T07:25:31Z

run external

eldenmoon · 2026-11-13T12:19:55Z

run external

eldenmoon · 2026-11-14T07:14:55Z

run external

eldenmoon · 2026-11-14T12:19:23Z

run external

eldenmoon · 2026-11-14T12:19:58Z

run external

eldenmoon · 2026-11-14T16:14:21Z

run external

eldenmoon · 2026-11-15T16:13:17Z

run external

eldenmoon requested a review from yiguolei as a code owner November 7, 2026 04:50

eldenmoon force-pushed the branch-variant-cast-nullable-41-20261107 branch from 694b6ea to 43ccf7d Compare November 7, 2026 05:04

eldenmoon force-pushed the branch-variant-cast-nullable-41-20261107 branch 2 times, most recently from 41d6057 to ce1ffe3 Compare November 7, 2026 06:52

eldenmoon mentioned this pull request Nov 7, 2026

branch-4.2: [fix](cast) Declare Variant casts to non-Variant targets nullable #70359

Open

eldenmoon added 4 commits November 12, 2026 10:11

eldenmoon force-pushed the branch-variant-cast-nullable-41-20261107 branch from ce1ffe3 to dde6111 Compare November 12, 2026 02:11

Conversation

eldenmoon commented Nov 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Fix

Validation

Compatibility and performance

Release note

Check List (For Author)

Update: explicit Variant check + NOT NULL-column regression

Uh oh!

hello-stephen commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Local pipeline review — ✅ PASS

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

hello-stephen commented Nov 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Nov 7, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

hello-stephen commented Nov 7, 2026

FE UT Coverage Report

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 7, 2026

Uh oh!

eldenmoon commented Nov 8, 2026

Uh oh!

eldenmoon commented Nov 8, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 12, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 13, 2026

Uh oh!

eldenmoon commented Nov 14, 2026

Uh oh!

eldenmoon commented Nov 14, 2026

Uh oh!

eldenmoon commented Nov 14, 2026

Uh oh!

eldenmoon commented Nov 14, 2026

eldenmoon commented Nov 7, 2026 •

edited

Loading